Background: Nowadays, computer-aided diagnosis systems are widely used in medicine. These systems could assist medical doctors in early correct diagnosis of diseases. The performance of such systems must be correctly assessed. In this paper, the performance criteria of these diagnosis systems are taken into account.Methods: The diagnosis error of such systems was estimated based on the gold standard data using measures such as Sensitivity, Specificity, Accuracy, Precision, Area Under curve ROC (receiver operating characteristic), F-measure, Matthews correlation coefficient , etc. The advantages and disadvantages of those criteria were also discussed. Then, the performance of two Coronary Artery Disease diagnosis systems was assessed. The statistically significant superior method was identified using the McNemar’s test.Findings: Since the analyzed dataset was balanced, the overall performance of the diagnosis methods was assessed using the accuracy measure. The accuracy of the methods was 84% and 86%, respectively. The entire systems were not reliable since Type I error (Alpha) was not less than 0.05. However, the second system had acceptable statistical power (>80%). The diagnosis performance of those systems was “very good” (0.8<AUC<0.9). None of the methods was better than the other.Conclusion: The performance of the diagnosis systems must be assessed using the proper methods and criteria. Using different suitable performance measures, it is possible to assess the diagnosis performance of such systems in details.